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19.  ABSTRACT  Continued. 

Modern  computer/microprocessor  based  control  and  surveillance  systems  have  evolved  into  two  basic  shipboard  "sys¬ 
tems";  - 

the  "combat  system"  comprised  of  radars,  sonar,  electronic  warfare  suites,  and  associated  control  processors,  and 

the  "engineering  control  system"  comprised  of  auxiliary,  electrical,  propulsion,  damage  control,  steering,  and  associated 
central  control  station  processors. 

Within  the  next  ten  years,  these  two  separate,  yet  interdependent  "systems"  are  expected  to  be  combined  over  a  common 
data  bus  into  one  shipboard  control  system.  Yet  the  overall  control  systems  of  today  have  been  designed  and  procured  by  accu¬ 
mulating  and  interfacing  the  equipments  and  subsystem  components  with  little  understanding  of  how  the  total  system  will  func¬ 
tion  under  various  operating  conditions,  workloads  and  degrees  of  casualties  or  battle  damage.  Further,  the  typical  system 
procurement  is  based  on  long  used,  and  often  outdated,  standards  and  specifications  defined  for  stand-alone  equipments  or 
subsystems  of  the  1950-1970’s.  The  result  has  been  that  new  control  systems  have  been  fielded  with  obsolescent  system  level 
control  because  of  the  inability  to  determine  how  the  introduction  of  new  or  evolving  technology  will  cross  the  procurement 
interface  and  affect  the  total  system  under  development.  Unlike  modern  data  networks  that  have  system  integrity  features  to 
enable  network  users  to  determine  that  the  network  is  passing  data,  the  individual  systems  with  the  built-in-test  (BIT)  and 
training  features  are  not  designed  to  perform  in  a  team  environment. 
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ABSTRACT 

Shipboard  systems,  subsystems,  and  equipments  are  not 
designed  to  optimize  overall  operational  performance  or  mission 
effectiveness.  The  problem  manifests  itself  through  system  level 
urder  utilization  of  potential  warfighting  capability  and 
operational  performance  inherent  in  the  highly  capable  sub-system 
and  equipment  components. 

This  is  considered  attributable  to  the  fact  that  our 
shipboard  crews  do  not  operate,  test,  train  on,  or  assess 
performance  of  well  defined  systems,  subsystems,  and  conponents 
with  carefully  integrated  operator  surveillance  and  control 
functions  that  enhance  system  level  operator  decision  making  and 
response  actions.  Rather  they  tend  to  employ  an  aggregation  of 
equipments  and  component  tests  and  training  to  perform  a  sub 
function  and  it  is  up  to  the  operators  to  assess  performance  and 
determine  the  operational  interaction  of  conponents.  The 
operational  use  of  the  "system"  and  assessment  of  responses  to 
sensor  inputs  are  relegated  to  training  and  all  too  often  it  is 
training  or  the  operator  himself,  rather  than  control  system 
design  that  is  considered  at  fault  when  the  system  or  operators 
are  incapable  of  cxping  with  an  operational  situation. 

The  integration  of  test,  training,  and  performance 
assessment  requirements  into  control  and  surveillance  system 
design  has  became  a  necessity  due  to  escalating  complexity,  costs 
of  duplicative  and  redundant  design,  equipment,  software  and  the 
pressing  need  to  provide  for  real  time  utilization  of  full  total 
ship  capability. 

The  integrated,  distributed  control  and  surveillance  systems 
of  today  provide  the  baseline  for  total  system  level  test, 
training,  arri  performance  assessment  as  well  as  the  intrinsic 
system  level  capability  to  meet  operational  performance 
requirements  if  included  in  preliminary  design. 

This  paper  discusses  recent  developments  in  overlapping 
requirements,  system  design,  functional  allocation,  and 
operational  implementation  of  Integrated  Test,  Training,  and 
Performance  Assessment  utilizing  a  distributed  microprocessor 
control  and  surveillance  system  design. 


TRAINING 


In  addition  to  equipment  and  computer  program 
considerations,  shipboard  system  designs  specifically  includes 
the  human  operator  as  part  of  the  operational  design  philosophy. 
Uius,  the  operators  timely  and  correct  response  to  stimuli  is 
essential  to  system  performance.  Realistic  total  system 
training  time,  underway  or  inport,  is  minimal.  The  attrition  of 
operator  and  team  skills  without  continuous  training  on  the 
systems  is  such  that  it  declines  to  the  point  that  even 
if  the  mechanical  and  electrical  components  are  operable,  the 
overall  system  performance  is  below  design  capability.  Just  as 
important  to  training  the  operators  and  decision  makers,  is 
training  them  realistically  and  recognizing  the  shortcomings  of 
their  abilities,  the  shortcomings  of  their  equipments  and 
computer  programs,  and  learning  hew  to  cope  with  the  conditions 
at  hand.  Constant  and  continuing  practice  and  assessment  of 
the  individuals,  subteams,  and  the  ship  as  a  team  entity 
becomes  vitally  important  for  ship  safety  in  peacetime  and 
survival  in  battle. 


PERFORMANCE  ASSESSMENT 


In  an  active  sense,  where  the  input  is  controlled, 
performance  assessment  is  a  comparison  of  the  actual  output  to  an 
expected  output  or  standard. 


Performance  assessment  can  also  be  passively  accomplished. 
In  this  case,  performance  assessment  is  more  of  a  surveillance  or 
monitoring  function.  For  exairple,  the  Combat  Information  Center 
Watch  Officer  and  Engineering  Officer  of  the  Watch  perform 
passive  performance  assessment  functions. 


Whether  active  or  passive,  performance  assessment  is  a 
surveillance  of  the  systems  vital  signs  and  an  analysis  of  the 
inpact  of  non-standard  events  on  the  operational  capability  of 
the  system.  System  performance  can  be  improved  with  knowledge  of 
the  systems  actual  state  through  system  control  design  features. 


INTEGRATED  TEST,  TRAINING,  AND  PERFORMANCE  ASSESSMENT 


To  test  and  assess  performance  at  the  shipboard  system 
level,  a  stimuli  can  be  injected  into  the  front  end  of  the 
cctmbat  systems  surveillance  systems  and  monitored  with 
surveillance  devices  and  techniques  through  each  component, 
equipment,  sub-system.  The  external  stimuli  causes  certain 
responses  by  the  equipment  and  computer  programs  of  the  combat 
system  which  in  turn  causes  certain  responses  by  the  equipment 
and  computer  programs  of  the  engineering  control  system.  This 
technical  approach  is  a  philosophy  best  described  as  "from 
the  outside-in"  as  contrasted  to  the  aggregation  of 
component,  equipment,  and  subsystem  "inside-out"  testing  at 
all  levels.  The  external  STIMULATION  causes  the  ship  to  be 
tested  and  assessed  as  one  entity.  Ihe  surveillance  radars, 
sonars,  electronic  warfare,  and  intelligence  suites  musi' 
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function,  but  they  in  turn  are  directly  dependent  on  the 
engineering  functions  of  the  ship  working  to  their  designed 
efficiency. 

As  the  external  stimulation  causes  the  combat  system  and  the 
engineering  system  equipments  and  computer  programs  to  react,  it 
additionally  causes  the  third  component  and  perhaps  the  cost 
important  component  of  the  "system",  the  people,  to  react.  The 
stress  test  causes  the  operators  to  form  habits,  thoughts,  and 
behavior  patterns  to  ccmplete  the  tests,  operate  their  equipments 
and  ccsiputer  programs,  and  cope  with  the  situation  as  the  tests 
are  run.  Thus  as  the  system  is  tested  and  the  equipment  and 
ccsiputer  programs  performance  is  assessed,  it  is  also 
accanplishing  training  and  performance  assessment  of  the  people. 
The  functions  of  test,  training,  and  performance  assessment  are 
so  intertwined  when  front  end  stimulation  is  induced  that  it 
is  logical  to  integrate  them  into  one  being.  Integrated  Test, 
Training,  and  Performance  Assessment  (ITT&PA) . 


INTEGRATED  TEST,  TRAINING,  AND  PERFORMANCE  ASSESSMENT  MODEL 

Figure  1,  respresents  a  model  of  the  principles  of 
Integrated  Test,  Training,  and  Performance  Assessment  (ITT&PA) . 


Figure  1  •  Integrated  Test,  Training,  &  Performance  Assessment  Model 


The  Scenario  Generation  and  Control  function  is  the  origin 
of  the  test  to  be  conducted.  The  Scenario  Generation  and  Control 
function  causes  the  stimuli  to  be  injected  into  the  equipment  and 
computer  programs  via  stimulation  /  simulation  techniques.  Since 
people  operate  and  control  the  equipments  and  computer  programs 
they  too  are  exercised.  That  is,  the  Scenario  Generation  and 
Control  function  is  test  control  and  it  is  training  control. 
Surveillance  devices  and  techniques  monitor  the  reaction  of  the 


hardware,  software,  and  people  and  feedback  the  test/training 
results  to  the  control  function.  A  comparison  of  the  known  input 
to  the  actual  output  versus  the  expected  output  allows  the 
performance  of  the  system  to  be  determined. 

CCMEAT  SYSTEM  EXAMPLES 

The  AN/USQ-93  (V)  Shipboard  Radar  Environmental  Simulator 
System  (RESS)  is  being  built  as  a  complement  to  the  New  Threat 
Upgrade  Anti-Air  Warfare  Combat  System  improvement  program. 
System  design  includes  a  Test,  Training,  Performance  Monitoring 
Controller,  radar  and  IFF  stimulators,  and  distributed 
processors.  The  si’stem  was  built  for  test  but  is  used 
extensively  for  training. 

The  AN/SQQ-89  On  Board  Trainer  (OET)  is  being  built  as  a 
sonar  suite  trainer  but  will  have  extreme  applicability  for  sonar 
stress  tests. 

The  Perry  Class  Pierside  Combat  System  Team  Trainer,  Device 
20B5  uses  a  scenario  generator/ controller,  sensor  and  weapon 
stimulation  and  simulation  techniques,  distributed  processing, 
and  fiber  optic  communication  connectivity  to  train  the  entire 
FFG— 7  Class  Combat  System.  Lessons  learned  indicate  that  even 
though  it  was  built  as  a  trainer,  signficant  amount  of  "system" 
tests  are  conducted. 

The  1HD-1  class  ship  is  built  with  the  AN/SSQ-91  (V)  combat 
Simulation  Test  System  (CSTS) .  A  test  system,  that  like  its 
predessor  currently  onboard  onboard  the  DDG-993  class  ship,  will 
also  train  the  combat  system  team.  The  architecture  of  the  LHD-1 
CSTS  is  illustrated  in  Figure  2. 


ENGINEERING  APPLICATION 

As  combat  systems  are  starting  to  implement  system  level 
test,  training,  end  performance  assessment  tools,  the  lessons 
learned  become  applicable  to  the  engineering  aspects  of  the  ship. 
Figure  3  illustrates  the  concept  of  ITT&PA  applied  to  the 
engineering  requirement  for  test,  training,  and  performance 
assessment  of  its  "system".  Although  the  concepts  of  machinery 
testing  and  performance  assessment  have  been  explored  and 
utilized  in  the  past,  for  example  Test  Evaluation  and  Monitoring 
System  (TEAMS) ,  Centralized  Automatic  Test  System  (CATS) ,  and 
Shipboard  Machinery  Performance  Monitoring  System  (SMFMS) ,  the 
integration  of  test,  training,  and  performance  assessment  for  the 
engineering  system  is  a  relatively  new  concept. 
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Figure  2  -  LHD-1  Combat  Simulation  Test  System  Architecture 


An  underlying  requirement  for  improved  continuity  of 
operation  of  shipboard  engineering  systems  will  be  the  ability  to 
monitor  the  sub-systems  exemplified  in  Figure  3.  Bui  It- in- test 
(BIT)  and  built- in-test-equipment  (BITE)  in  the  individual  sub¬ 
systems  are  in  many  cases  already  very  close  to  being  the 
"stimulator"  in  the  Integrated  Test,  Training,  and  Performance 
Assessment  concept. 

SURVEILLANCE 

Surveillance  or  monitoring  is  needed  to  provide  continuous 
and  immediate  information  on  engineering  plant  and  component 
status,  level  of  performance,  and  to  detect  equipment  failures 
and  predict  impending  failures  so  that  corrective  action  can  be 
taken  by  a  supervisory  control  system.  Concurrently,  the 
information  available  on  machinery  components  will  reduce 
maintenance  requirements  by  providing  the  capability  for 
maintenance  on  demand  rather  them  on  a  regular  calendar  schedule 
basis. 
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Figure  3  *  Concept  ITT&PA  for  the  Engineering  System 


The  types  of  input  versus  output  measurements  that  ITT&PA 
could  address  in  the  engineering  system  include  surveillance  / 
monitoring  of  pressure,  motion,  flew,  temperature,  force,  torque, 
and  changes  in  the  same  parameters  caused  by  operator  actions. 


For  example,  a  large  fraction  of  mechanical  measurements  is 
concerned  with  pressure,  particularly  in  propulsion  systems, 
process  control  systems,  and  ordnance  systems.  Pressure  is  the 
actual  quantity  required  in  many  instances;  hewever,  in  other 
cases  the  desired  parameter  is  inferred  fresa  pressure 


measurements.  Urns  flew  may  be  measured  in  terms  of  differential, 
pressure  across  a  deliberate  flow  obstruction  or  in  terms  of  the 
force  exerted  against  a  deflection  plate.  In  the  case  of  notion, 
measurements  need  to  be  made  either  of  the  motion  of  objects  in 
inertial  space  or  of  the  motion  of  certain  parts  of  objects  with 
respect  to  other  parts.  The  type  of  measurement,  the  amplitude 
and  frequency  response  requirements ,  as  well  as  certain 
evironmental  considerations  'usually  dictate  the  particular  aspect 
of  motion  sensed  and  the  type  of  sensor  used.  Motion  sensors 
convert  linear  and  angular  displacement,  and  their  time 
derivatives:  velocity,  accleration,  and  jerk,  to  electrical 
signals.  Each  of  the  motion  parameters  may  be  directly  sensed 
and  converted  in  one  or  more  steps  to  the  desired  electrical 
signal,  or  the  desired  signal  ray  be  derived  by  differentiation 
or  integration  from  a  signal  transduced  from  one  of  the  related 
motion  parameters. 

Surveillance  of  output  as  compared  to  input  leads  to  the 
performance  assessment  of  the  individual  sub-systems.  Controlled 
input  via  test/training  controller  and  surveillance  of  the  output 
parameters  enables  the  system.  to  be  assessed  as  a  whole. 
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It  is  desirable  that  the  engineering  system,  like  the  ccmbat 
system,  be  tested  and  the  people  who  operate  them,  be  trained, 
to  provide  two  primary  levels  of  information.  Ihe  first  level 
is  to  assess  the  "well  being"  of  the  equipment,  computer 
programs,  and  people  and  inform  decision  "makers  with  that 
information  tailored  to  his  needs.  The  second  level  is  to  aid  in 
the  location  of  failed  components  for  the  technicians. 


The  engineering  officer  of  the  watch  is  responsible  for  the 
safe  and  proper  operation  of  the  engineering  system.  In  that 
regard  ITT&PA  can  provide  him  from  a  distributed  and  continuously 
update  data  base,  assigned  mission  responsibility  as  directed 
from  the  combat  system.,  commanding  officer,  and/or  officer  of  the 
deck.  With  this  information  the  engineering  officer  of  the  watch 
can  employ  ships  propulsion,  electric  power,  auxilary  systems, 
interior  communications,  etc.  ITT&PA  can  provide  him  with  his 
engineering  availability,  limitations,  and  recommended 
alternatives  for  system,  employment  based  upon  its  current  state. 
ITTaPA  can  provide  similar  information  to  the  damage  control 
officer  who  has  access  to  the  data  base  through  distributed  data 
buses.  Integrated  Test,  Training,  and  Performance  Assessment  can 
aid  in  fault  isolation,  trend  analysis,  and  decision  aids. 


The  ccrabat  system  components,  equipments,  and  subsystems 
are  physically  located  throughout  the  ship,  as  are  the 
cccponents,  equipments,  and  subsystems  of  the  engineering  system. 
The  combat  system  and  the  engineering  system,  hewever,  must 
merge  to  form  an  n-dimensional  complex  machine,  the  ship. 
Computer  technology  today  is  such  that  distributed  processing 
bectnes  a  viable  solution.  Effective  and  efficient  communication 


connectivity  between  the  two  major  systems  is  required  to  ensure 
total  ship  control  for  daily  operation,  control  for  safetv,  and 
control  during  battle  for  survivability.  The  data  buses  of  today 
provide  the  baseline  for  application  of  the  integrated  test, 
training,  and  performance  assessment  concept.  Figure  4 
represents  a  conceptual  integrated  test,  training,  and 
performance  assessenent  architecture  with  distributed  control 
workstations. 


Performance  Assessment  Data  Base 


Test/Training 
Control  &  Processing 


Surveillance 


Combat  System 

Equipment,  Computer  Programs,  &  People 


Data  Bus 


Engineering  System 

Equipment,  Computer  Programs,  &  People 


EOOW 


Surveillance 


Test  /  Training 
Control  &  Processing 


Performance  Assessment  Data  Base 


ASWE  =  Anti-Submarine  Warfare  Evaluator 

CO  =  Commanding  Officer 

DCA  =  Damage  Control  Assistant 

EOOW  r  Engineering  Officer  of  the  Watch 


STIM  =  Stimulation 

TAO  =  Tactical  Action  Officer 

XO  =  Executive  Officer 


Figure  4  -  Integrated  Test,  Training,  end  Performance  Assessment 
with  Distributed  Control  and  Surveillance  Design 


itt&pa  objective 

lbs  objective  of  Integrated  Test,  Training,  and  Performance 
Assessment  is  to  provide  shipboard  decision  rakers  and 
maintenance  personnel  with  a  means  of  assessing  in  real  time  the 
"state  of  health"  of  the  ship  to  optimize  overall  operational 
system  performance. 

PERFORMANCE 

Knowledge  that  the  system  is  not  functioning  to  its  maximum 
design  capability  becomes  essential  not  only  during  battle  but 
during  everyday  operational  use  for  safety  and  efficiency.  For 
example,  what  is  impact  on  the  ship  if  a  power  supply  to  the 
surveillance  radar  is  degraded?  What  is  the  impact  if  the 
watch  supervisor  in  firerocsn  number  two  faints  due  to  heat 
exhaustion  caused  by  inadequate  air  conditioning?  Who  will 
replace  the  tactical  action  officer  if  he  is  severely  injured 
from  a  missile  explosion?  Hew  will  the  electrical  load  be  re¬ 
distributed  and  hew  fast? 

The  combat  system  operators  and  decision  rakers  must  knew 
in  real  time  the  constantly  changing  status  of  the  engineering 
system.  They  must  knew  the  limitations  of  their  systems  at  all 
times  in  order  to  exploit  the  resultant  capabilities.  The 
propulsion  system  operators,  technicians,  and  decision  makers 
must  know  the  ever  changing  combat  system  status  in  order  to 
initiate  repairs,  maintenance,  and  shifts  in  load  centers  to 
exploit  its  maximum  capability  upward  into  the  combat  system. 
Information  format  and  utility  is  different  for  the  technicians, 
operators,  and  decision  makers.  The  readiness  status  information 
must  therefore  be  tailored  to  best  support  the  users  of  the 
information  in  the  shortest  time  possible.  With  the  diversity  of 
information  required,  the  diversity  of  the  elements,  components, 
and  sub-sy stems,  the  spacial  distances  between  the  equipments  and 
personnel ,  and  state  of  the  art  technology  a  distributed 
architecture  is  appropriate . 

COST  SAVINGS  . 

While  the  US  Navy  is  recognizing  the  need  for  system  level 
testing  and  for  system  level  training  only  recently  is  it 
beginning  to  recognize  the  overlap  in  test  and  training 
methodologies.  From  a  cost  standpoint,  integrated  test, 
training,  and  performance  assessment  offers  an  opportunity  to  pay 
only  once  for  a  more  complete  integrated  test  and  training 
capability.  This  in  itself  is  reason  for  integrating  the  closely 
related  functions  of  test  and  training. 

The  efforts  ongoing  in  combat  system  test  /  training  have 
all  been  add  on  capabilities.  Recognition  of  the  need  for 
system  level  test  and  training  has  come  after  the  operational 
equipments  have  been  funded  and  developed.  lessons  learned 
indicate  that  had  the  test  /  training  capabilities  been  disigned 
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into  the  prime  item  equipments  cost  savings  could  have  been 
realized  not  only  in  research  and  development  but  in  life  cycle 
logistics  costs.  Additionally,  the  add  on  stimulators  and 
simulators  have  caused  space  and  weight  tradeoffs.  The  Device 
20B5  for  example  comes  in  semi-truck  and  is  therefore  not  an 
asset  available  at  sea.  Advances  in  today's  technology  in  the 
area  of  very  high  speed  integrated  circuitry,  local  area 
networks,  fiber  optics,  lap  top  carputers  all  make  feasible  the 
concept  of  embedding  the  ITT&PA  capabilities  and  techniques  into 
prime  item  equipments  at  the  design  stage. 

RESEARCH 

The  operational  use  of  combat  systems  in  response  to 
stimulation  and  simulation  resulting  have  resulted  in  lessons 
learned  that  indicate  all  too  often  it  is  training  or  the 
operator  himself,  rather  than  control  system  design  that  is 
considered  at  fault  when  the  system  or  operators  are  incapable  of 
coping  with  an  operational  situation.  While  research  and 
development  facilties  are  slcwly  moving  forward  toward 
discovering  these  lessons  through  experimentation,  recognition 
that  the  ship  must  operate  as  one  entity  which  includes  the 
engineering  system  is  significantly  lacking. 

Data  that  would  be  collected,  processed,  and  distributed  to 
the  users  should  be  maintained.  The  data  offers  insight  into  the 
availability  and  maintainability  of  the  conplex  system  as  a 
whole.  Trend  analysis  of  the  operational  systems  offers  the 
opportunity  to  identify  problem  areas  and  feed  that  information 
back  into  the  acquisition  requirements  definition  process. 

CONCLUSION 

Increases  in  the  performance  characteristics  and  level  of 
automation  of  surface  ship  systems  emphasize  the  need  for 
extensive  and  continuing  training,  as  well  as  continuous  and 
automated  system  performance  measurement.  Moreover,  the 
complexity  of  current  systems  and  lack  of  total  system  training 
cpportunties  make  it  imperative  that  operators  and  teams  have 
the  opportunity  to  train  in  realistic  simulated  environments  in 
order  to  develop  and  maintain  required  performance  levels. 

It  is  also  essential  that  ccmnanders  have  an  accurate  and 
up-to-date  operability  status  of  the  systems  under  their  control. 
With  the  increasing  attention  paid  to  cost  effectiveness,  the 
high  cost  of  dedicated  training  systems,  and  stringent 
limitations  on  import  crew  availability  for  training,  there  is  a 
critical  requirement  for  an  embedded  Integrated  Test,  Training, 
and  Performance  Assessment  capability  capable  of  being  used  at 
sea  or  in  port. 

As  the  overall  equipment  and  software  requirements  of 
training  and  performance  assessment  functions  are  substantially 
similar,  significant  cost  savings  can  accrue  from  a  combined 
effort. 


It  is  clear  that  further  progress  in  Integrated  Test, 
Training,  and  Performance  Assessment  will  be  closely  linked  to 
the  evolution  of  control  and  surveillance  system  designs  and 
their  associated  data  buses. 
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